Research Methods in Human Sciences
September 5, 2023
05.09.2023 12.00 - 14.00: Introduction into LLM analysis and notebook environment for practical exercise
07.09.2023 12.00 - 14.00: Practical exercises with text data and group feedback
Sources: Bender et al. “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big”, Ted Chiang in New Yorker, Simon Willison
Source: Luccioni, Alexandra Sasha, and Anna Rogers. “Mind Your Language (Model): Fact-Checking LLMs and Their Role in NLP Research and Practice.”
Source: Ziems, Caleb et al. “Can Large Language Models Transform Computational Social Science?”
Typical workflow:
| Training your own model | Supervised text categorisation with LLMs | Zero/Few-shot annotation with LLMs | ||
|---|---|---|---|---|
| Models used | Model chosen for task | LLM of choice | Typically LLM that can be prompted with instructions | |
| Preparing model | Training model on your own dataset | Finetuning existing LLM | Preparing and testing prompts that instruct model | |
| Data requirements | Training and validation dataset (potentially large dataset necessary) | Training and validation dataset | Just validation dataset |
hateful speech: Messages that
Finetuning is a process where an pre-trained model goes through a smaller training process to perform a specific task.
model = AutoModelForSequenceClassification.from_pretrained(
"TurkuNLP/bert-base-finnish-uncased-v1"
)
training_args = TrainingArguments(
"nethate", evaluation_strategy="epoch", logging_steps=30
)
metric = load_metric("accuracy")
trainer = Trainer(
args=training_args,
model=model,
train_dataset=train_dataset,
eval_dataset=eval_dataset,
compute_metrics=compute_metrics,
)
trainer.train()
finetuned_finbert = pipeline(
model=model, tokenizer=tokenizer, task="sentiment-analysis", return_all_scores=True
)
Goal: A “boring”, boilerplate model
Source: https://github.com/AleksiKnuutila/nethate_classifier/blob/master/train.py
Source: Pangakis et al. “Automated Annotation with Generative AI Requires Validation”
An example of annotating text: Assessing the amount of time passing in literary passages.
Source: Underwood, “Why Literary Time is Measured in Minutes”
Read the following passage of fiction. Then do five things.
1: Briefly summarize the passage.
2: **Reason step by step to decide how much time is described in the passage. If the
passage doesn't include any explicit reference to time, you can guess how much
time the events described would have taken**. Even description can imply the
passage of time by describing the earlier history of people or buildings. But
characters' references to the past or future in spoken dialogue should not count
as time that passed in the scene. Report the time using units of years, weeks,
days, hours, or minutes. Do not say zero or N/A.
3: If you described a range of possible times in step 2 take the midpoint of the
range. Then multiply to convert the units into minutes.
4: Report only the number of minutes elapsed, which should match the number in
step 3. Do not reply N/A.
5: Given the amount of speculation required in step 2, describe your certainty
about the estimate--either high, moderate, or low.
The passage follows: <omitting this to save space>
Source: https://tedunderwood.com/2023/03/19/using-gpt-4-to-measure-the-passage-of-time-in-fiction/
1: A detective wakes up 'the next morning,' looks out a window for an undefined
time, drinks (and presumably needs to make) two cups of coffee, then shaves and
showers and gets dressed before stepping out his front door and seeing a car.
2: **Making coffee, showering, and getting dressed take at least an hour. There's
some ambiguity about whether to count the implicit reference to yesterday
(since this is 'the next morning') as time elapsed in the passage, but let's
say no, since yesterday is not actually described. So, an hour to 90 minutes.**
3: 1.25 hours have elapsed. Multiplying by 60 minutes an hour that's 75 minutes.
4: 75 minutes.
5: Low confidence, because of ambiguity about a reference to the previous day.
“Watching the model think out loud about each passage actually gave me a better sense of how results were produced than I could extract from my own records of human research in 2017.”
Source: Tracy, “A Phronetic Iterative Approach to Data Analysis in Qualitative Research”
Slides at http://knuutila.net/llm_methods
:: {.notes} - Spirit of exercises: above all, try out something that is useful for you - Here are some resources that you can work with :::